Online-Academy
Look, Read, Understand, Apply

Data Analytics

Advanced Data Analytics

To perform statistics like t-test, normal distribution, ANOVA we have to install statsmodels. We can install multiple libraries using single pip install command as follows:

pip install numpy pandas scipy statsmodels matplotlib

NumPy (short for Numerical Python) is the foundational open-source Python library used for scientific computing, mathematical operations, and working with large multi-dimensional arrays. It is the backbone of Python's data science ecosystem, powering major libraries like Pandas, Scikit-learn, and TensorFlow.

  • pearonr: Correlation Analysis (Pearson Correlation). This measures the strength of the relationship between two variables.
  • shaprio: Normality Test (Shapiro-Wilk Test). Checks whether data follows a normal distribution.
  • linregress: Linear Regression Analysis. Predict one variable from another.
  • ttest_ind: Independent t-Test. Compare the means of two independent groups.
  • ttest_rel: Paired t-Test. Used when the same participants are measured twice.
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import pearsonr,linregress,ttest_ind,f_oneway,ttest_rel,shapiro

# Sample data: creating numpy array.
#x = np.array([10, 20, 30, 40, 50])
#y = np.array([15, 25, 35, 45, 60])

x = [10, 20, 30, 40, 50]
y = [15, 25, 35, 45, 60]

# Correlation
r, p = pearsonr(x, y)

print("Correlation coefficient (r):", round(r, 4))
print("P-value:", round(p, 4))

if p < 0.05:
    print("Significant correlation.")
else:
    print("No significant correlation.")

Linear Regression Analysis. Predict one variable from another.

import numpy as np
from scipy.stats import linregress

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1,2,3,4,5,6])
y = np.array([2,4,5,4,5,7])

# Regression
result = linregress(x, y)

print("Slope:", result.slope)
print("Intercept:", result.intercept)
print("R-squared:", result.rvalue**2)
print("P-value:", result.pvalue)

# Prediction
y_pred = result.slope*x + result.intercept

print("\nPredicted values:")
print(y_pred)
#Plot Regression Line
#import matplotlib.pyplot as plt

plt.scatter(x, y)
plt.plot(x, y_pred)
plt.xlabel("X")
plt.ylabel("Y")
plt.title("Linear Regression")
plt.show()

ANOVA (One-Way ANOVA): Compare the means of three or more groups.

from scipy.stats import f_oneway

# Sample groups
group1 = [20,22,19,24,25]
group2 = [30,32,31,29,33]
group3 = [26,28,25,27,29]

F, p = f_oneway(group1, group2, group3)

print("F-statistic:", round(F,4))
print("P-value:", round(p,4))

if p < 0.05:
    print("Groups are significantly different.")
else:
    print("No significant difference.")

Independent t-Test Compare the means of two independent groups.

from scipy.stats import ttest_ind

group1 = [70,72,68,75,74]
group2 = [80,79,82,81,78]

t, p = ttest_ind(group1, group2)

print("t-statistic:", round(t,4))
print("P-value:", round(p,4))

if p < 0.05:
    print("Means are significantly different.")
else:
    print("Means are not significantly different.")

Paired t-Test Used when the same participants are measured twice.

from scipy.stats import ttest_rel
before = [60,65,62,68,70]
after  = [65,67,66,70,75]

t, p = ttest_rel(before, after)

print("t-statistic:", round(t,4))
print("P-value:", round(p,4))

Normal Distribution Generate normally distributed data and visualize it.

import numpy as np

"""
Key Parameters Breakdown
loc=50: The mean (center) of the distribution. 
The average of all 1,000 numbers will be very close to 50.
scale=10: The standard deviation of the distribution. This dictates the spread of the data. 
For a standard normal curve:~68% of the numbers will fall between 40 and 60 (± 1 standard deviation).
~95% will fall between 30 and 70 (± 2 standard deviations).
~99.7% will fall between 20 and 80 (± 3 standard deviations).
size=1000: The output shape of the array. 
It will return a 1D array containing exactly 1,000 random data points.
"""
# Generate random data
data = np.random.normal(loc=50, scale=10, size=1000)

print("Mean =", np.mean(data))
print("Standard Deviation =", np.std(data))

plt.hist(data, bins=30)
plt.title("Normal Distribution")
plt.xlabel("Value")
plt.ylabel("Frequency")
plt.show()

Normality Test (Shapiro-Wilk Test) Checks whether data follows a normal distribution.

from scipy.stats import shapiro
data = [10,12,15,16,18,19,20,21,22,25]
stat, p = shapiro(data)

print("Statistic:", stat)
print("P-value:", p)

if p > 0.05:
    print("Data is normally distributed.")
else:
    print("Data is NOT normally distributed.")

Correlation Analysis Using Pandas

import pandas as pd

data = {
    "Math":[60,70,80,90,85],
    "Science":[65,75,82,88,90],
    "English":[70,72,75,80,78]
}

df = pd.DataFrame(data)

print(df.corr())